Towards Distributed Machine Learning in Shared Clusters: A Dynamically-Partitioned Approach

摘要

Many cluster management systems (CMSs) have been proposed to share a singlecluster with multiple distributed computing systems. However, none of theexisting approaches can handle distributed machine learning (ML) workloadsgiven the following criteria: high resource utilization, fair resourceallocation and low sharing overhead. To solve this problem, we propose a newCMS named Dorm, incorporating a dynamically-partitioned cluster managementmechanism and an utilization-fairness optimizer. Specifically, Dorm uses thecontainer-based virtualization technique to partition a cluster, runs oneapplication per partition, and can dynamically resize each partition atapplication runtime for resource efficiency and fairness. Each applicationdirectly launches its tasks on the assigned partition without petitioning forresources frequently, so Dorm imposes flat sharing overhead. Extensiveperformance evaluations showed that Dorm could simultaneously increase theresource utilization by a factor of up to 2.32, reduce the fairness loss by afactor of up to 1.52, and speed up popular distributed ML applications by afactor of up to 2.72, compared to existing approaches. Dorm's sharing overheadis less than 5% in most cases.